SPECIFICATION 



TO ALL WHOM IT MAY CONCERN: 

Be it known that we, Erik M. Geidl, a citizen of the 
United States, residing at 1641 171st Avenue NE, Bellevue, 
Washington 98008, Robert A. Dain, a citizen of the United 
States, residing at 11524 133rd Avenue NE, Redmond, Washington 
98052, and Donald D. Karlov, a citizen of the United States, 
residing at 14227 192nd Avenue NE, Woodinville, Washington 
98072, have invented a certain new and useful ELECTRONIC INK 
AS A SOFTWARE OBJECT of which the following is a 
specification . 



ELECTRONIC INK AS A SOFTWARE OBJECT 



FIELD OF THE INVENTION 

The present invention relates generally to computing 
devices, and more particularly to handwritten input used with 
computing devices. 

BACKGROUND OF THE INVENTION 

Contemporary computing devices allow users to enter 
handwritten words (e.g., in cursive handwriting and/or printed 
characters), characters and symbols (e.g., characters in Far 
East languages) . The words, characters and symbols can be 
used as is, such as to function as readable notes and so 
forth, or can be converted to text for more conventional 
computer uses. To convert to text, for example, as a user 
writes strokes representing words or other symbols onto a 
touch-sensitive computer screen or the like, a handwriting 
recognizer (e.g., trained with millions of samples, employing 
a dictionary, context and/or other rules) is able to convert 
the handwriting data into dictionary words or symbols. In 
this manner, users are able to enter textual data without 
necessarily needing a keyboard. 

Applications have been developed that know how to handle 
this handwritten input, including sending the user input to a 



recognizer at appropriate times. These applications provide 
the user with various features related to both the handwritten 
ink as written and the text as recognized. To this end, these 
applications maintain the handwritten data (electronic ink) in 
association with any recognized text in their own internal 
data structures. 

While such an electronic ink application provides 
valuable features to users, many other things that most users 
intuitively associate with ordinary text or rich text are not 
provided. For example, if a document having the electronic 
ink input is saved as a file that is later reopened, only the 
recognized text may be available, i.e., the handwriting part 
of recognized data may not be kept. Another significant 
problem is that text is easily understood by other 
applications, while the handwriting data is not. Thus, even 
if the handwriting data is preserved with a document, the 
handwritten data can only be understood by the application 
into which it was entered. For example, when the user wants 
to send a document having electronic ink therein to another 
computer (e.g., as an e-mail attachment), the other computer 
may not have a copy of that application, and thus cannot 
interpret the other application's ink data. 

Still other features associated with text, such as cut 
and paste operations, are easily performed with text data. 



The application that receives text (via the paste operation) 
can easily insert it into existing text in an appropriate 
manner. For example, the text is automatically aligned with 
any existing text on its line, it can be easily reformatted/ 
searched, and so forth. This is not possible with handwritten 
input and contemporary application programs. 

In general, electronic ink does not behave like text in a 
number of significant ways. Notwithstanding, it is becoming 
apparent that users want electronic ink to be more 
interchangeable and/or like text with respect to transferring 
it among computing devices and/or different programs, editing 
functions, and so on. 

SUMMARY OF THE INVENTION 

Briefly, the present invention provides electronic ink as 
a software object, such as a COM (Component Object Model) 
object, thereby associating functionality with ink data, by 
which the large base of applications that can generally deal 
with embedded objects can benefit from electronic ink. The 
object's functionality can handle operations such as rendering 
the ink data, sizing to match text, communicating with a 
recognizer to obtain recognition data, and provide other 
valuable ink-related features to an application that is not 
necessarily coded with any knowledge of electronic ink. 



Because the ink data is maintained as an object, the data is 
automatically persisted as part of any document into which the 
ink object is embedded. In general, via the electronic ink 
object, electronic ink substantially approaches many of the 
5 behaviors normally inherent in text data. 

A suitable implementation of an electronic ink object 
format is provided, including the communication, rendering and 
sizing functionality that enables ink-unaware applications to 
benefit from electronic ink. Additional functionality also 
Jj;10 may be provided, in the form of callable methods, to enable 
5 applications coded with the knowledge of such electronic ink 
g objects ("ink-aware" applications) to further provide 
jl electronic ink-related features. For example, once the ink 
3 has been sent to a recognizer, the electronic ink object 
; f 15 maintains the recognition results, generally in the form of a 
*f list of ranked text word alternates, possibly including 

probability data. When an ink- aware application is performing 
a text search in a document, the application can call a method 
of an electronic ink object to access its maintained 
20 alternates, to determine whether the handwritten ink matches a 
text search term. In this manner, handwritten ink appears to 
be searched as if it was text in the document. Other methods 
enable an ink-aware application to format the ink, such as to 
change its color, bold it and/or italicize it, change its 
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relative size or position, and perform other functions 
normally associated with text. The application need not 
duplicate this functionality or deal with how the ink data is 
arranged, but rather simply can call an appropriate method 
with any appropriate parameter values. 

Other advantages will become apparent from the following 
detailed description when taken in conjunction with the 
drawings, in which: 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIGURE 1 is a block diagram representing an exemplary 
computer system into which the present invention may be 
incorporated; 

FIG. 2 is a block diagram generally representing 
components for handling electronic ink handwritten into a 
computer system and creating electronic ink objects therefrom 
in accordance with one aspect of the present invention; 

FIG. 3 is a representation of the contents of example 
handwritten input and how it may be separated into electronic 
ink objects in accordance with one aspect of the present 
invention; 

FIG. 4 is a block diagram generally representing an 
electronic ink object useable by applications in accordance 
with one aspect of the present invention; 



FIG, 5 is a representation of an application accessing 
the data and functionality of electronic ink objects via 
interfaces in accordance with one aspect of the present 
invention; 

FIG. 6 is a representation of example handwritten input 
inserted into text; 

FIG. 7 is a representation of the example handwritten 
input of FIG. 6 resized by functionality of the electronic ink 
objects for matching the electronic ink data when rendered to 
the text's size in accordance with one aspect of the present 
invention; 

FIG. 8 is a representation of example handwritten input 
being adjusted by functionality of electronic ink objects to 
enhance their displayed appearance, in accordance with one 
aspect of the present invention; 

FIG. 9 is a representation of various data fields 
maintained in an example electronic ink object in accordance 
with one aspect of the present invention; and 

FIG. 10 is a representation of various interfaces 
provided in an example electronic ink object for invoking the 
methods thereof in accordance with one aspect of the present 
invention. 



DETAILED DESCRIPTION 

EXEMPLARY OPERATING ENVIRONMENT 

FIGURE 1 illustrates an example of a suitable computing 
system environment 100 on which the invention may be 
implemented. The computing system environment 100 is only one 
example of a suitable computing environment and is not 
intended to suggest any limitation as to the scope of use or 
functionality of the invention. Neither should the computing 
environment 100 be interpreted as having any dependency or 
requirement relating to any one or combination of components 
illustrated in the exemplary operating environment 100, 

The invention is operational with numerous other general 
purpose or special purpose computing system environments or 
configurations. Examples of well known computing systems, 
environments, and/or configurations that may be suitable for 
use with the invention include, but are not limited to, 
personal computers, server computers, hand-held or laptop 
devices, tablet devices, multiprocessor systems, 
microprocessor-based systems, set top boxes, programmable 
consumer electronics, network PCs, minicomputers, mainframe 
computers, distributed computing environments that include any 
of the above systems or devices, and the like. 

The invention may be described in the general context of 
computer-executable instructions, such as program modules, 



being executed by a computer. Generally, program modules 
include routines, programs, objects, components, data 
structures, and so forth, that perform particular tasks or 
implement particular abstract data types. The invention may 
5 also be practiced in distributed computing environments where 
tasks are performed by remote processing devices that are 
linked through a communications network. In a distributed 
computing environment, program modules may be located in both 
local and remote computer storage media including memory 

10 storage devices. 

With reference to FIG. 1, an exemplary system for 
implementing the invention includes a general purpose 
computing device in the form of a computer 110. Components of 
the computer 110 may include, but are not limited to, a 

15 processing unit 120, a system memory 130, and a system bus 121 
that couples various system components including the system 
memory to the processing unit 120. The system bus 121 may be 
any of several types of bus structures including a memory bus 
or memory controller, a peripheral bus, and a local bus using 

20 any of a variety of bus architectures. By way of example, and 
not limitation, such architectures include Industry Standard 
Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, 
Enhanced ISA (EISA) bus, Video Electronics Standards 
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Association (VESA) local bus, and Peripheral Component 
Interconnect (PCI) bus also known as Mezzanine bus. 

The computer 110 typically includes a variety of 
computer-readable media. Computer-readable media can be any 
available media that can be accessed by the computer 110 and 
includes both volatile and nonvolatile media, and removable 
and non-removable media. By way of example, and not 
limitation, computer-readable media may comprise computer 
storage media and communication media. Computer storage media 
includes both volatile and nonvolatile, removable and non- 
removable media implemented in any method or technology for 
storage of information such as computer-readable instructions, 
data structures, program modules or other data. Computer 
storage media includes, but is not limited to, RAM, ROM, 
EEPROM, flash memory or other memory technology, CD-ROM, 
digital versatile disks (DVD) or other optical disk storage, 
magnetic cassettes, magnetic tape, magnetic disk storage or 
other magnetic storage devices, or any other medium which can 
be used to store the desired information and which can 
accessed by the computer 110. Communication media typically 
embodies computer-readable instructions, data structures, 
program modules or other data in a modulated data signal such 
as a carrier wave or other transport mechanism and includes 
any information delivery media. The term "modulated data 



signal" means a signal that has one or more of its 
characteristics set or changed in such a manner as to encode 
information in the signal. By way of example, and not 
limitation, communication media includes wired media such as a 
5 wired network or direct-wired connection, and wireless media 
such as acoustic, RF, infrared and other wireless media. 
Combinations of the any of the above should also be included 
within the scope of computer-readable media. 

The system memory 130 includes computer storage media in 

10 the form of volatile and/or nonvolatile memory such as read 
only memory (ROM) 131 and random access memory (RAM) 132. A 
basic input/output system 133 (BIOS), containing the basic 
routines that help to transfer information between elements 
within computer 110, such as during start-up, is typically 

15 stored in ROM 131. RAM 132 typically contains data and/or 
program modules that are immediately accessible to and/or 
presently being operated on by processing unit 120. By way of 
example, and not limitation, FIG. 1 illustrates operating 
system 134, application programs 135, other program modules 

20 136 and program data 137, 

The computer 110 may also include other removable/non- 
removable, volatile/nonvolatile computer storage media. By 
way of example only, FIG. 1 illustrates a hard disk drive 141 
that reads from or writes to non-removable, nonvolatile 
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magnetic media, a magnetic disk drive 151 that reads from or 
writes to a removable, nonvolatile magnetic disk 152, and an 
optical disk drive 155 that reads from or writes to a 
removable, nonvolatile optical disk 156 such as a CD ROM or 
other optical media. Other removable/non-removable, 
volatile/nonvolatile computer storage media that can be used 
in the exemplary operating environment include, but are not 
limited to, magnetic tape cassettes, flash memory cards, 
digital versatile disks, digital video tape, solid state RAM, 
solid state ROM, and the like. The hard disk drive 141 is 
typically connected to the system bus 121 through a non- 
removable memory interface such as interface 14 0, and magnetic 
disk drive 151 and optical disk drive 155 are typically 
connected to the system bus 121 by a removable memory 
interface, such as interface 150. 

The drives and their associated computer storage media, 
discussed above and illustrated in FIG. 1, provide storage of 
computer-readable instructions, data structures, program 
modules and other data for the computer 110. In FIG. 1, for 
example, hard disk drive 141 is illustrated as storing 
operating system 144, application programs 145, other program 
modules 146 and program data 147. Note that these components 
can either be the same as or different from operating system 
134, application programs 135, other program modules 136, and 



program data 137. Operating system 144, application programs 
145, other program modules 146, and program data 147 are given 
different numbers herein to illustrate that, at a minimum, 
they are different copies. A user may enter commands and 
5 information into the computer 2 0 through input devices such as 
a tablet (electronic digitizer) 164, a microphone 163, a 
keyboard 162 and pointing device 161, commonly referred to as 
mouse, trackball or touch pad. Other input devices (not 
shown) may include a joystick, game pad, satellite dish, 

10 scanner, or the like. These and other input devices are often 
connected to the processing unit 120 through a user input 
interface 160 that is coupled to the system bus, but may be 
connected by other interface and bus structures, such as a 
parallel port, game port or a universal serial bus (USB) . A 

15 monitor 191 or other type of display device is also connected 
to the system bus 121 via an interface, such as a video 
interface 190. The monitor 191 may also be integrated with a 
touch-screen panel 193 or the like that can input digitized 
input such as handwriting into the computer system 110 via an 

20 interface, such as a touch-screen interface 192. Note that 
the monitor and/or touch screen panel can be physically 
coupled to a housing in which the computing device 110 is 
incorporated, such as in a tablet-type personal computer, 
wherein the touch screen panel 193 essentially serves as the 
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tablet 164. In addition, computers such as the computing 
device 110 may also include other peripheral output devices 
such as speakers 195 and printer 196, which may be connected 
through an output peripheral interface 194 or the like. 

The computer 110 may operate in a networked environment 
using logical connections to one or more remote computers, 
such as a remote computer 180. The remote computer 180 may be 
a personal computer, a server, a router, a network PC, a peer 
device or other common network node, and typically includes 
many or all of the elements described above relative to the 
computer 110, although only a memory storage device 181 has 
been illustrated in FIG. 1. The logical connections depicted 
in FIG. 1 include a local area network (LAN) 171 and a wide 
area network (WAN) 173, but may also include other networks. 
Such networking environments are commonplace in offices, 
enterprise-wide computer networks, intranets and the Internet. 

When used in a LAN networking environment, the computer 
110 is connected to the LAN 171 through a network interface or 
adapter 170. When used in a WAN networking environment, the 
computer 110 typically includes a modem 172 or other means for 
establishing communications over the WAN 173, such as the 
Internet. The modem 172, which may be internal or external, 
may be connected to the system bus 121 via the user input 
interface 160 or other appropriate mechanism. In a networked 



environment/ program modules depicted relative to the computer 
110, or portions thereof, may be stored in the remote memory 
storage device. By way of example, and not limitation, FIG. 1 
illustrates remote application programs 185 as residing on 
memory device 181. It will be appreciated that the network 
connections shown are exemplary and other means of 
establishing a communications link between the computers may 
be used. 

ELECTRONIC INK OBJECTS 

The present invention is primarily directed to electronic 
ink, which in general corresponds to a set of X, Y coordinates 
input by a user, and additional state information such as an 
"up" or "down" state. Notwithstanding, it will be appreciated 
that the present invention is applicable to virtually any type 
of user input that corresponds to words or symbols that can be 
mixed with and/or recognized as text, such as speech data. 
Thus, although for purposes of simplicity the present 
invention will be described with reference to handwriting 
input and display thereof, and especially examples of English 
cursive handwriting, the present invention should not be 
limited in any way to handwritten input and/or by the examples 
used herein. 



As a further simplification, the user may be considered 
as entering ink input via a pen-tip (cursor) that writes on a 
tablet-like device, such as the touch-screen panel 193, Note 
that this may not be literally correct for all devices and/or 
5 in all instances. For example, some devices such as a mouse 
or a pen capture device do not have a real, physical tablet 
and/or pen-tip. For such devices, a virtual tablet may be 
assumed. In other instances, electronic ink may be generated 
by an application program or other software, in which event 
^SlO the tablet and pen-tip may both be considered to be virtual, 
p Electronic ink is often more complex than simple X, Y 
4S coordinate data, and may have additional properties associated 
Ol therewith. For example, tip pressure, angle data, writing 
) 3: speed, an d so on may be maintained as part of the ink data. 
^15 In general, electronic ink may be maintained in any 

suitable form with any type of accompanying data. 
In accordance with one aspect of the present invention, 
electronic ink is maintained as an object, having both data 
and associated functionality, whereby object-compatible (e.g., 
20 capable of embedding COM objects) applications can interface 
with the objects to provide electronic ink features. This 
provides benefits to applications that are themselves not 
otherwise capable of working with electronic ink data, 
generally referred to herein as "ink-unaware" applications or 
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the like. For example, a spreadsheet document that knows 
nothing of electronic ink can open a document with an 
electronic ink object embedded therein, and automatically 
obtain many of the benefits of electronic inking, as described 
5 below, including displaying the ink to a user. Because the 

electronic ink can be embedded as an object into documents and 
the like, the electronic ink object may be freely pasted into 
documents and transferred among various applications, with its 
full integrity maintained, regardless of the application's 

nlO ink-related capabilities. Note that because the electronic 

y 

3 ink is an embedded object, it is persisted to non-volatile 

f:: storage (e.g., streamed to disk) without requiring the 

P application to store the data. Moreover, the document with 

3 the embedded ink can be transferred to another computer, such 

^ 15 as via an e-mail attachment. 

"-*,. 

f Applications that are coded with an understanding of 

electronic ink objects ("ink-aware" applications) can further 
take advantage of many of the features and functions offered 
by the electronic ink object, without needing to duplicate the 
20 object's functionality. For example, such an application can 
invoke various methods that enable an ink-aware application to 
format the ink in ways that are very similar to text 
formatting, e.g., to change the ink's attributes such as its 
bold state, italicized state and/or color, and its relative 
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size and position. Other methods enable an ink-aware 
application to perform other functions related to the 
electronic ink that are normally associated with text. For 
example, once the ink has been sent to a handwriting 
5 recognizer, the electronic ink object maintains the 

recognition results, generally in the form of a list of ranked 
text word alternates, and possibly including probability data. 
When an application is performing a text search, the 
application can call a method of an electronic ink object to 

fflO access its maintained alternates to determine whether the 

handwritten ink matches a text search term. In this manner, 

g handwritten ink appears to be searched as if it was text in 

|1; the document. 

3; Turning to FIG. 2 of the drawings, various components of 

^15 the computing system 100 or the like are represented when the 
if computing system 100 is configured for receiving handwritten 

input, in accordance with one aspect of the present invention. 
In FIG. 2, as a user enters data such as by handwriting data 
on the touch screen panel 193, (and/or by another input 
20 mechanism 200 such as the keyboard 162) , input circuitry 
and/or other components 202 pass the information to the 
operating system 134. Depending on the state of the system 
and/or where the screen is contacted, the user input may be 
converted to text or treated as handwritten data. For 
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example, when the screen displays a representation of a 
keyboard, and that keyboard is touched, the coordinates may be 
converted to an ASCII or Unicode character, or an editing 
command, and sent to an appropriate component (e.g., a word 
5 processor application having focus) to use. Alternatively, if 
the user is writing in a handwriting input area, the input may 
correspond to coordinate (stroke data) . As is understood, the 
handwritten data may be in the form of coordinates, and may 
include other information, e.g., pressure data, timing data 
SlO and so forth. The user input is typically reflected back to 
3 the user in some way, e.g., via display output circuitry / 
p components 2 04 and/ or via other output mechanisms 206 (e.g., 

J1 as a spoken word) ♦ 

3 To determine the meaning of the user input, the operating 

'2 15 system may include or otherwise be associated with an input 
5 f handling mechanism 208 that may includes an ink / text handler 
210 to differentiate between text and handwriting data. In 
general, ASCII / UNICODE characters and editing commands are 
passed directly to the application 212 that currently has 
20 input focus, for maintaining within an appropriate text 

container 214i-214 m or the like, while stroke data is first 
pre-processed, as described below. Note that if stroke data 
is received, the system may be arranged to pass stroke data 
directly to an application program if such an application is 

- 18 - 



capable of handling it, however for purposes of the present 
example, the electronic ink will be pre-processed (e.g., by 
the input handling mechanism 208) prior to providing the 
electronic ink to an application. 

More particularly, and in accordance with one aspect of 
the present invention, prior to providing electronic ink to 
the application, the handwritten input will be processed into 
electronic ink objects 216i-216 n . In one implementation, the 
ink objects comprise COM (Component Object Model) objects in 
the form of an "Embedded Content Object," as described in the 
reference entitled "Inside OLE, second edition, " Kraig 
Brockschmidt (Microsoft Press), hereby incorporated by 
reference. Embedded Content Objects are simplified versions 
of ActiveX® objects, with some advantages, e.g., embedded 
Objects can be lazily loaded, and displayed using a cached 
visual representation. In general, COM is a standard for, 
among other things, brokering software objects within and 
between software applications, and COM objects are widely 
supported across many applications. COM objects are defined 
to have certain consistent behaviors, whereby applications 
that support COM objects in general will automatically 
support, at least to some minimal degree, the electronic ink 
object of the present invention when implemented as a COM 
object. CORBA (Common Object Request Broker Architecture) is 



another such standard infrastructure, and thus it can be 
readily appreciated that the electronic ink object of the 
present invention may be incorporated into COM or CORBA type 
object technology in general. 

To create the electronic ink objects, the input handling 
mechanism 208 includes or is otherwise associated with the 
necessary ink object creation module 218 that creates an 
object instance for the raw data. In general, for purposes of 
convenience, each word that is written may correspond to one 
object (e.g./ in English, French or German implementations), 
or each written symbol such as a character may correspond to 
one object (e.g., in Japanese, Chinese, Korean 
implementations), although other units (e.g., character pairs, 
parts of a character, sentences, lines and so forth) may 
alternatively be used, and even mixed (e.g., some objects 
represent sentences, others words, and still others 
characters) in a given implementation. To this end, the ink 
object creation module 218 may communicate with a handwriting 
recognizer (recognition engine) 220 to determine how the 
received ink should be separated, e.g., into words. 
Alternatively, not all computing devices that can input 
electronic ink have a handwriting recognizer, and thus the ink 
object creation module 218 may include its own logic and rules 
to determine how the ink should be separated into words. Such 



rules can be based on criteria such as the horizontal 
separation between strokes, the timing data between a pen-up 
and a pen down, and other known criteria. Alternatively, an 
application that is aware of electronic ink may be configured 
to control word breaks and the like as desired by the 
application / user thereof for viewing purposes and the like* 
In any event, one preferred implementation provides one object 
per word, since word processors and the like typically treat a 
word as an atomic unit when performing document formatting, 
word wrapping and so forth. Notwithstanding, although the 
description and examples herein generally refer to one object 
per word, it should be understood that an electronic ink 
object can be provided per character, symbol, phrase, clause, 
sentence, paragraph or any other grouping of words (such as a 
fixed number of words or characters, a written line and so 
forth) , with appropriate data therein as desired for combining 
separate objects into one word and/or separating one object 
into multiple words. 

By way of example, FIG, 3 shows electronic ink 
handwritten into a user interface (or program) 300 that is 
suitable for receiving electronic ink data. The handwritten 
input for the words x 'This is hand written" is separated into 
four objects 301-304, which may or may not have resulted from 
the ink having been passed to a recognizer. As represented in 



FIG. 3, each handwritten word having an associated object 301- 
304 is encompassed by a imaginary dashed box (not actually 
visible to the user) . In general, a user presently has two 
choices for entering electronic ink handwriting data, the 
5 "lined" mode represented in FIG. 3, in which the user writes 
upon a guide with baseline, midline and ceiling line. 
Alternatively, the user can write anywhere on the screen, 
(except of course, to areas corresponding to commands such as 
one that would close the application) , such as generally 
b if; 10 represented in FIG 6. Note that other data capture modes are 
Q feasible, e.g., by providing a special window for handwriting 

; p: input. Indeed, virtually any user interface mode is feasible, 

!rt and the present invention is not limited to the example modes 

M described herein. 

"2 15 In the lined mode example, the positions of the guide 

Yz lines may be included in the ink data to help calculate the 
rendering size, and to assist with handwriting recognition. 
Guide line data is not available in full-screen mode, (one 
possible alternative user interface mode) , but can be estimated 
20 by the object, as generally described below. Regardless of the 
data entering mode, data representing the position of the mouse 
or stylus (e.g., from the system message WM_MOUSEMOVE) are 
collected and sent to an electronic ink object. Note that the 
data can be sent to the object as a whole, or in multiple, 
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smaller batches as the handwriting data is collected. Further, 
note that the ink data in maintained at a very high 
resolution, which preserves the original resolution of the 
input device. For example, touch-sensitive (input) screens 
typically have much higher resolution than the display device 
(output) . When rendered on the screen, the original data 
points are scaled into screen coordinates. The original high 
resolution data is retained, even when the electronic ink 
object is passed through devices that do not have the ink 
object functionality (e.g., in a library of functions, 
described below) , or passed through applications which do not 
natively understand electronic ink objects. 

Returning to FIG. 2, the ink object creation module 218 
(which may be considered part of the object's functionality or 
separate therefrom) also may process the ink data in other 
ways, e.g., by multiplying by X and Y factors to normalize the 
ink to a standard size, increasing or decreasing the number of 
coordinates to match a system limitation (e.g., a 16-bit 
coordinate system) , compressing the ink data, smoothing the 
ink data in some way (although smoothing may instead be done 
when rendering an image of the ink), and so forth. For 
example, one of the output data formats (discussed below) may 
comprise the Windows® Metafile format, which is limited to 16- 
bit coordinate values, whereby entered handwriting data may be 
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checked and scaled if necessary to keep all coordinate data 
values within the range of a 16-bit signed integer (less than 
32, 767) . 

In the current implementation, the input handling 
5 mechanism 208 also calls the handwriting recognizer 220, if 
available, and receives a list of the ten top alternates 
(candidates) which it provides to the electronic ink object 
(e.g., 216i) to store for later reference and use, e.g., for 
rendering as text, searching, and so on. Note that the 
:~fl0 electronic ink object 21 6 X may also call the handwriting 
2 recognizer 220 directly, including after creation and when 
IS embedded in a document. Further, note that the present 

n invention is not dependent on handwriting recognizers generally 
3; or on any particular handwriting recognizer. As such 

15 recognition generally will not be described herein, however one 
J- suitable recognizer is able to return a list of alternates when 

provided with ink data, along with other information, including 
baseline information, as described below. Handwriting 
recognizers are generally described in U.S. Patent Nos. 
20 5,313,527 and 5,467,407. 

Note that because handwriting recognition may be 
performed at a later time, or performed again (e.g., with a 
different recognizer), the objects may contain information 
that allow the originally entered ink data to be reconstructed 
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from the ink data maintained in the objects* For example, as 
described below with particular reference to FIGS. 8 and 9, 
any X and Y factors used in normalization are maintained with 
the object, and also an object may contain identifiers or 
pointers to its previous and next objects, if any, so that the 
ink data may be provided to the recognizer as it was 
originally entered by the user, and not as initially processed 
by the ink object creation module 218, object functionality 
and/or handwriting recognizer 220. 

In keeping with the present invention, each electronic 
ink object (e.g., 216!) is arranged to persist its ink data to 
non-volatile storage 218 in association with the document or 
the like into which it is embedded, so that the object data 
will automatically be part of with its containing document. 
To this end, the electronic ink objects may be serialized 
(stored on disk) via a standard set of functions designed to 
work in any application that supports COM objects. Note that 
the format of the ink is controllable, and can be compressed, 
or uncompressed. By adhering to this standard, the electronic 
ink object can be serialized as part of the native data format 
of many applications. Such functionality works automatically, 
with no modifications to existing applications required. 
Because the document contains the electronic ink object, the 
electronic ink object essentially moves with the document. 



Thus, for example, if the document is e-mailed to another 
computer, the electronic ink object is transferred with the 
document . Note that if the ink data has been sent to a 
recognizer 220, recognition information including at least one 
5 text alternate may be part of the data persisted with the 

object, along with other data such as baseline data returned 
by the recognizer that helps in aligning the object with text, 
as described below. 

FIG. 4 represents an ink object, such as the ink object 
pJlO 216i of FIG. 2, used by various types of application programs 
^ 400, 402 and 404. In general, various types of applications 
irj may be in use, including an application type 4 00 that is not 

J| ink-aware, but is capable of embedding objects. Such 
3 applications 400 are able to open a document with objects 

a 15 embedded therein, or later have objects embedded into an 
=* already open document by pasting them via a clipboard 408, and 
call the object to render an image of the ink. Note that 
while the present invention provides substantial benefits with 
text-oriented applications such as word processing and 
20 spreadsheet programs, the present invention also allows 

drawing programs or the like to embed the ink objects, whereby 
the drawings programs can at least have the handwritten data 
displayed as part of its drawing. 
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If a document includes an electronic ink object (or 
objects) , and the functionality associated with an electronic 
ink object is available to the system on which the application 
400 is running, (e.g., as a dynamic link library or DLL and/or 
5 in the operating system), the application 400 can obtain many 
benefits of electronic ink objects. Note that if the DLL or 
the like is not available, the application will display the 
object as a bitmap or other image, whereby the handwriting 
data may appear as it was last rendered. For example, the 

2 10 icon displayed by the application to represent the presence of 
; S that object may be a bitmap of the handwritten data as last 

g rendered. Note that with the electronic ink object as an 

f| embedded object, even if the DLL or the like that provides the 

3 functionality for that object is not present on a particular 
j 15 machine, the integrity of the ink object data is preserved 

when transferred to and from that machine. This is also true 
when the document embedding the object is opened with 
applications that are not ink aware, i.e., the object's 
integrity is preserved. Thus, the object data is fully 
20 available for later use when the functionality is present on a 
machine, or when an ink aware application later accesses it, 
regardless of where it has been. 

In a typical situation, the electronic ink object 
functionality will be available to the object, whereby among 
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other functions, the object will render itself. In one 
implementation, the object will render itself in a manner that 
aligns and sizes to the surrounding text. By way of example 
of how a text-based application might embed an electronic ink 
5 object, FIG. 5 shows an application 500 that maintains a text 
container 502 with pointers or the like that link electronic 
ink objects 504, 506 to the text. Note that the text 
container 500 may comprise a container object or other similar 
data structure. In any event, the text container 500 or 
210 similar data structure maintains the objects' relative 
n positions within the text data. As the text container is 
g filled in and edited, the application displays the text, and 
fl when it reaches an electronic ink object, calls a method on 
3 the electronic ink object to render itself, which may include 

™J- 15 sizing itself to the surrounding text, as described below. 
The application then causes the remaining text to be 
appropriately displayed relative to the object. For example, 
with electronic ink, the application may output the next set 
of text characters to the location that follows a rectangular 
20 area specified by the object. In this manner, even if an 

application is ink-unaware, the handwritten words essentially 
appear to the user right along with the text, with each text 
and ink word correctly positioned in the document. 
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FIGS. 6, 7 and 8 provide an example of the object's 
resizing functionality in more detail. Note that in FIG. 6, a 
user has written (e.g., directly onto the application window 
at a blinking cursor 600 or the like) words that are 
5 noticeably larger than their surrounding text. Upon 

subsequent rendering, e.g., after the text is edited or the 
user has stopped writing for a sufficient time, the displayed 
representation of the electronic ink is resized to better 
match the surrounding text, as generally represented in FIG. 
'?10 7. Note that as shown in FIG. 4, the object allocates or 
f± includes some memory 410 for working with the rendering data, 
that is, the ink as stored in the object is not changed, but 
in instead the object constructs and manages data points 

p corresponding to the image to display. 

'"H. 

i ; j 15 To resize the ink, as generally shown in FIG. 8 (using 

]4 different words for illustration) , the ink is considered 

relative to a baseline 800 and a midline (dashed line) 802. 
Significantly, for alignment, the baseline of the displayed 
ink should match the baseline of any surrounding text. Note 
20 that the baseline is not the bottom of the characters, but 
rather corresponds to the bottom of characters that do not 
have descenders, e.g., the baseline of the word "together" 
corresponds the bottom of the "to" and "ether" letters and not 
to the bottom of the "g" letter. The recognizer 220 may 
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determine baselines, and if so, this information is maintained 
with the object's data. For electronic ink data that has not 
been recognized, the ink object contains logic that determines 
a baseline reasonably well, e.g., by looking for reasonably 
consistent y-minima over the x-coordinate length of a word, 
with compensation for any detected writing angle. Other logic 
can factor in the overall size of a word, and even consider 
the baselines of other recently written or surrounding 
objects . 

In addition to aligning the baseline of the handwritten 
image or images to that of the text, the size of the displayed 
ink can be increased or decreased relative to the surround 
text. To this end, the ink coordinates can be multiplied by a 
factor as needed to match the baseline-to-midline region 804 
to that of the text's baseline to midline region. Note that 
the midline is the top of characters without ascenders, e.g., 
the midline of the word "be" corresponds to the top of the w e" 
letter and not the top of the "b" character , since the x> b" 
character has an ascender. Again, the functionality in the 
object can contain the necessary logic to reasonably determine 
a midline (such as via consistent y-maxima over the x- 
coordinates) if not provided by a recognizer. 

Further, and as generally represented in FIG. 8, if the 
handwritten word has overlarge ascenders or descenders, then 



the outlying data points associated with those artifacts is 
scaled (e.g., in a nonlinear manner) to shrink them while 
reasonably retaining the shape of the ink. This is done so 
that the displayed ink will not overwrite text on lines above 
5 and below it. Thus, any ink extending above a ceiling line 
(top limit) 806 is scaled to fit into an ascender region 808, 
while any ink extending below a floor line (lower limit) 810 
may be similarly scaled to fit into a descender region 812. 
To this end, any suitable function such as linear 
JflO multiplication function on a sliding scale (e.g., that adjusts 

a 

; ==l coordinates further from the baseline more and coordinates 

g closer to the baseline less) may be employed to fit the ink 

ft into the appropriate region. Note that this scaling 

3 essentially handles extreme ascenders and/or descenders 

j 15 without resizing the entire word based on such extremes, 

: f whereby the baseline-to-midline region of the word reasonably 

matches the text's baseline-to-midline region, which generally 

looks better. 

Lastly, it should be noted that other data manipulation 
20 functions may be employed that are generally directed to 

improving the appearance of the rendered image. For example, 
a smoothing function may be employed prior to display. In one 
implementation, the data points are sent to a Bezier curve 
generation algorithm, and the resulting Bezier data points are 
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passed to the Microsoft® Windows® GDI method PolyBezier for 
final rendering. Such Bezier curves are a common mathematical 
construct used for drawing smooth curves, and may be used as a 
rendering mechanism with electronic ink objects to improve the 
appearance of digitized handwriting, which tends to be somewhat 
jagged due to the coarseness of the data capture hardware and 
unsteadiness of the user's hand. Drawing may also employ the 
anti-aliasing APIs available in GDI+. 

Returning to FIG. 4, another application type 404 may be 
one that is incapable of embedding objects. Such applications 
404 may be able to have text pasted into them, however, and 
thus one way in which the ink object 21 61 may be used by such a 
limited application, is to put corresponding text (if 
previously recognized) onto the clipboard 408 (or the like) 
that is used for cutting and pasting operations. Note that 
such clipboards normally maintain a variety of formats, and 
for example, as one of its supported formats, text, the 
clipboard 408 may be arranged to receive the best (most- 
probable) text word alternate when copying the ink object. If 
so, when pasting to the text-only application 404, the 
application 404 receives that alternate as text. Note, 
however, that such an application does not have the object 
embedded therein, whereby if a document of that application is 
opened by another application, only the text is present. 
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A third type of application is one that is specifically 
aware of electronic ink objects, and can utilize their 
associated functionality. Such applications can call an 
object's various exposed methods for specific purposes, such 
as to format the ink (e.g., to bold it, italicize it, change 
its color) , resize the displayed image, and so forth. To this 
end, the user may select objects (and possibly surrounding 
text) by a click-and-drag operation. Then, for example, if 
the user chooses to italicize the ink, then the ink aware 
application calls a formatting method of the object, whereby 
the object skews its data points in the rendering data 410 
(FIG. 4) to the right to provide the italicized look. A 
general listing exemplifying various methods available to 
applications, along with a brief description thereof, is 
provided as Appendix A. 

Note that in addition to text and Windows® metafile 
formats, other common data formats may be supported for use 
across many applications, e.g., via the clipboard 408. The 
Electronic Ink object can render itself in several of these 
formats, as requested by the rendering application via standard 
interfaces. Such supported formats in one current 
implementation are Enhanced Metafile, Windows® Metafile, 
Windows clipboard format, Windows bitmap, and plain text. 
Thus, the same electronic ink object may render itself 
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differently, depending on the display capabilities of the 
application. 

Searching is a fundamental element of text processing, and 
electronic ink objects that have been recognized support 
searching operations. To this end, the electronic ink object 
maintains a list of top alternate words returned by the 
handwriting recognition engine 220, and provides access to the 
alternate words via an interface. This allows ink-aware 
applications (that know to call the electronic ink object 
interfaces) to implement string searching across electronic ink 
objects. Note that an application is not limited to exact text 
searching with the top alternates or alternates, but instead, 
because the application has access to each of the alternates, 
and not just the top one, virtually any matching scheme 
desired by an application with respect to the alternates may 
be employed. For example, one scheme may look for a 
percentage of matching characters, with a user-determined 
threshold percentage. Other variations include giving 
different weight to certain characters, and/or factoring in 
the relative number of syllables (returned by some recognizers 
and thus maintainable with the object) and/or the relative 
lengths of the words. The probabilities of alternates may be 
used as a factor in the determination, and other factors and 
possible tests may be used by a given application. 
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In addition to the maintained ink data, data is 
maintained that allows the object to recreate the handwritten 
input as originally entered. By way of example FIG. 8 shows 
one such situation wherein further ink processing including 
recognition may be desirable. In FIG. 8, the handwritten 
input may be considered as one word -- together — , two words, 
— to gather --, or three words — to get her — . Thus, if a 
user selects this handwritten data and later requests 
recognition thereof, such as for the first time or by a 
different recognizer, the object data cannot simply be sent to 
the recognizer without risking errors caused by an earlier 
word-break estimation . 

To recreate the handwritten input, the object maintains, 
for each object, the normalization data needed to undo any 
normalization used for that object. Further, the object keeps 
identifying information about previous and next objects, if 
any, so that the "un-normalized" ink can be recombined with 
its surrounding ink. For example, if in FIG. 8 three objects 
were used to represent "to," "get" and "her" words, a single 
set of ink will be sent to the recognizer. 

FIG. 9 shows some of the data fields that may be 
maintained in an electronic ink object's data 900, including 
some header information that, for example, contains the 
offsets of the various fields, the identifiers of any previous 
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and next objects, if any, and the normalization factors. Note 
that FIG, 9 is illustrative only, and the data fields may be 
in any suitable order, additional fields may be present, and 
other fields may be removed. The following tables sets forth 
some information that may be maintained as part of an ink 
object's data: 



typedef enum e num I NKME T R I C_ FL AG S 
{ 

IMF_FONT_SELECTED_IN_HDC = 0x000 00001, 
IMF_ITALIC = 0x00000002, 
IMF_BOLD = 0x00000004 
} INK_METRIC_FLAGS ; 

IMF_FONT_SELECTED_IN_HDC 

tells the ink object whether it should use its own internal 
INKMETRIC font parameters, or query the hardware device 
context (hdc, supplied in the drawing call) for font data 

IMF_ITALIC 

Tells whether or not the ink should be rendered in italics 
mode 

IMF_B0LD 

Tells whether or not the ink should be rendered in bold mode. 



typedef enum enumGetCandidateFlags 
{ 

TCF_ALLOW_RECOGNITION = 0x00000001, 
TCF_FORCE_RECOGNITION = 0x00000002 
} GET DANDIDATE FLAGS ; 



TC F_ALLOW_RECOGNI T I ON 

This flag specifies that if a top candidate from the 
recognized alternates list is requested, but no alternatives 
are present in the object's alternate list, then the object 
should invoke the recognition engine to generate an alternate 
list 



T C F_FORCE_RE COGN I T I ON 

This flag specifies that the object should invoke the 
recognition engine to generate an alternate list even if 
there is already an alternate list currently available, 

typedef struct taglNKMETRIC { 

INT iHeight; // Ink height 

INT iFontAscent; // Assent height 

INT iFontDescent; // Descent height 

DWORD dwFlags; // ink metric flags defined above 

COLORREF color; // Color 

} INKMETRIC, *PINKMETRIC; 



iHeight 

Specifies the overall height of the font 



iFontAscent 

Specifies the distance from the font baseline to the font 
"ceiling" 

iFontDescent 

Specifies the distance from the font baseline to the font 
"floor" 

(iFontAscent + iFontDescent = iHeight) 
dwFlags 

Contains the values from INK_METRIC_FLAGS, defined above 
color 

Defines the RGB color value in which this ink object is to be 
rendered. 
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Other internal data structures may include: 



// StrokeHeader 

typedef struct tagSTROKEHEADER 
{ 

UINT cHeaderSize; // Number of bytes in header 
U1NT cStreamSize; // Number of bytes in stream 
UINT cAlternatesOf f set; 
UINT cGuideOffset; 

UINT cStrokeOf f set; // Starting position of stroke 

// data 

UINT nID; 
UINT nPrevID; 
UINT nNextID; 
DWORD dwFlags; 
RECT rectBounds; 



} STROKEHEADER; 



/ / Content flags 
// Size of ink in TIP (device 
// independent) coordinates 



This set of data is written into the serialization 
stream for use as an index when reading the stream. 



// Stroke 

typedef struct tagSTROKE 
{ 



UINT cPoints; 
BOOL fUpStroke; 
POINT Appoints; 
UINT *pPressure; 
UINT *pTime; 
UINT *pAngle; 
} STROKE; 



// Number of data points 

// 0 = normal; 1 = upstroke 

// Point array 

/ / Pressure array 

// Time array 

// Angle array 



This data structure contains the data points associated 
with one stroke. 
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// StrokeList 

typedef struct tagSTROKELIST 
{ 

UINT cStrokes; // Number of strokes 

STROKE Upstrokes; // Stroke array 
} STROKELIST; 

Contains the stroke data structures associated with this 
object 



// Ink Flags 

enum 

{ 

LOSSY 

LOSSLESS = 
lossless 

XYPOINTS = 
PRESSURE = 
TIME 

DS ANGLE = 



} 



0x00000001, 
0x00000002, 

0x00000004, 
0x00000008, 
0x00000010, 
0x00000020 



// Stream is compressed: lossy 

// Stream is compressed: 

// Data set contains XY points 

// Data set contains pressure 

// Data set contains time 

// Data set contains angle - 

// ANGLE is used by richink 



Defines flags for data compression and stroke contents. 



/ / Guide data type 0 
typedef struct tagINKGUIDE_0 
{ 

LONG nMidline; 
LONG nBaseline; 
} INKGUIDE_0; 

Defines the position of the midline and baseline 
relative to the ink strokes • 



FIG. 10 shows some of the interfaces to the methods of an 
electronic ink object 1000 as described herein and set forth 



in Appendix A. Note that the functionality of these methods 
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is not physically contained in the object (such as generally 
represented in FIGS . 4 and 5), but rather is in a shared 
library of functions such as a DLL, as is known with objects 
in general. 

5 In addition to these listed methods and data, additional 

functions and/or data may be provided to enhance the object's 
usability and provide additional features with respect 
thereto. For example, a user may right-click on a rendered 
object and, via user interface functionality of the object, 
;S;10 receive a user interface 412 (FIG. 4) with a number of 
5. selectable choices. Note that the user interface functionality 
g may be associated with the object, whereby a user can directly 
ft invoke certain functions even when working in an ink unaware 
3 application. One such choice may be to toggle the object 

; j 15 between its handwritten appearance and its recognized text, 

y 

- possibly allowing for selection of an alternate from the list 
maintained therewith. The text may be displayed (e.g., 
slightly) differently in the application to indicate to the 
user that it not simply ordinary text, but is backed by an 
20 object. 

Another function that may be provided allows users to 
merge multiple objects into one object and/or separate one 
object into multiple objects. Such functions enable the 
strokes in an ink object or between ink objects to be added 
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to, modified, deleted, merged, copied, extracted, transformed, 
cropped and split. For example, as shown above with respect 
to FIG. 8, a user may want the three words "to get her" to be 
a single object "together" for editing purposes and the like. 
A merge function may be manually initiated by the user or may 
be automatically performed, such as if the user selects 
"together" as an alternate. To this end, the ink data from 
the "get" and "her" objects may be copied into the "to" 
object, after re-computing for any normalization and the like, 
the alternates rearranged, the "next" object data copied from 
the last ("her") object, and so forth, followed by the "get" 
and "her" objects being deactivated or discarded. 

Similarly, a user can separate one object into another, 
so that, for example, a user is not required to have two words 
such as "to" and "gather" word wrap as a unit, be searched 
incorrectly, and otherwise act as a unit, simply because an 
initial estimate determined that only one word was entered. 
Again, such a separation function may be automatically invoked 
in response to a user selecting a two-word alternate. 
Further, a user may wish to edit the ink, such as to format 
one part of a word and not another, whereby separation into 
multiple objects would enable such a feature. To this end, a 
new object is created from the other, with appropriate linking 



identifiers modified in each, ink data of the other removed 

from each, and so forth. 

Still another function that may be provided involves 

alternate recognizers. For example, if the word was 
5 recognized with one (e.g., U.S. English) recognizer, but the 

current user wants to use a different (e.g., British) 

recognizer, the user can invoke one by an appropriate user 

interface, (e.g., provided via a right click). The object can 

then call the recognizer. 
"ilO Other features may be provided by the application to 

enable multiple objects to be treated as a whole. For 
,S example, a user may select and highlight an entire paragraph, 
ill with three different objects therein. If the user chooses to 
O format the entire paragraph, the application can separately 

1^ 15 call the appropriate method or methods of each object for this 

purpose. 

In addition, other functions may be globally performed 
across multiple objects without the application separately 
calling each object. One way in which this may be 
20 accomplished is to have the objects arranged to call other 
objects that are grouped therewith for global functions. 
Another way in which this may be accomplished is to have an 
entity such as a container object maintain a global list of 
object identifiers (a list of IUnknown PTRS) , whereby for 
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global functions, the container object may be called which in 
turn calls each of the object listed therein. 

Moreover, because in the present invention ink is 
arranged as an object, preferably a COM object, the ink may be 
transmitted live over a network, such as by using distributed 
object technology or the like. Remote collaboration using an 
electronic ink object is possible, e.g., a user can 
electronically handwrite notes onto another user's document, 
and vice versa. Two computing devices may be linked in some 
way, and use the ink object live, e.g., a user may write on a 
pocket-sized personal computer and have the resultant 
electronic ink object transferred over a wireless link to a 
desktop computer. 

As can be seen from the foregoing detailed description, 
there is provided an electronic ink object, that may be 
embedded into a document, and preserved with the resolution 
and accuracy of the ink maintained, even if the ink is scaled 
or transformed for display purposes. An application that 
supports embedded objects can have the ink object render 
itself, and thus need not directly draw a representation of 
the ink or otherwise deal with ink data. Ink-aware 
applications can further call the ink object's functionality 
to operate on the ink, such as for formatting or searching 
purposes, again without directly dealing with the ink data. 



The electronic ink object provides functionality that enables 
electronic ink to approach the many features associated with 
text, and further is arranged to integrate with text data in a 
manner that is visually appealing and intuitive to users that 
are familiar with text data operations. 

While the invention is susceptible to various 
modifications and alternative constructions, certain 
illustrated embodiments thereof are shown in the drawings and 
have been described above in detail. It should be understood, 
however, that there is no intention to limit the invention to 
the specific form or forms disclosed, but on the contrary, the 
intention is to cover all modifications, alternative 
constructions, and equivalents falling within the spirit and 
scope of the invention. 



